feat: the languagecontainer module to share data and functions with language containers by suztomo · Pull Request #2489 · googleapis/librarian

suztomo · 2025-10-06T18:42:13Z

The languagecontainer module and packages

This change introduces github.com/googleapis/librarian/languagecontainer module which consists of the following packages:

bazel: the functions for Bazel-file parsing
cmd: the entry point function that parses command line arguments
message: the data definition of the messages between the CLI and the language container (using the existing ones). The replace github.com/googleapis/librarian => ../ line in containers/go.mod allows us to reuse the data definition outside the containers folder.
execv: the execve child process execution
protoc: the protoc command wrapper

(Cody shared me that in Go, we should avoid using a generic package name such as "util". https://go.dev/blog/package-names#package-names says "They are often simple nouns". The package names above follow the guidelines.)

The module name "languagecontainer" is from the convention (no word separator) I found in the modules in https://github.com/googleapis/google-cloud-go. It's descriptive than the original idea of "containers" (plural).

They are mostly from the packages from https://github.com/googleapis/google-cloud-go/tree/main/internal/librariangen. I renamed request to message (because it includes the response). I didn't include postprocessor (the logic is specific to google-cloud-go), testdata (the data is specific to google-cloud-go), build, generate, and release (they are the implementation of Go language containers commands).

Language Container's View

A language container declares the dependency as below:

module cloud.google.com/go/internal/librarian/java_container

go 1.25.0

require github.com/googleapis/librarian/languagecontainer 06b8d3305e95a7b9efe3e6641d45488e421dca21

where cloud.google.com/go/internal/librarian/java_container is language container's package name (irrelevant to this PR) and 06b8d3305e95a7b9efe3e6641d45488e421dca21 is a SHA of https://github.com/googleapis/librarian repository. The go mod tidy command replaces the SHA with the data + SHA (e.g., v0.0.0-20251007191759-06b8d3305e95).

With cmd.LanguageContainerMain, the main function of a language container will look like this:

func generateFunc(ctx context.Context, generateFlags *cmd.GenerateFlags) {
	// TODO: Implement the generate function. Read generateFlags that tells the context
	// locations in the filesystem as per https://github.com/googleapis/librarian/blob/main/doc/language-onboarding.md#generate
}

func main() {
	cmd.LanguageContainerMain(os.Args,
		cmd.LanguageContainerFunctions{
			Generate: generateFunc,
		})
}

Tests

I added a test case as cmd_test.go to demonstrate that the argument parsing tests.

suztomo@suztomo98:~/librarian/languagecontainer$ go test ./...
?       github.com/googleapis/librarian/languagecontainer/bazel [no test files]
ok      github.com/googleapis/librarian/languagecontainer/cmd   0.007s
ok      github.com/googleapis/librarian/languagecontainer/execv 0.021s
?       github.com/googleapis/librarian/languagecontainer/message       [no test files]
?       github.com/googleapis/librarian/languagecontainer/protoc        [no test files]

Because we test argument parsing in this module, the language container implementations will not need to test their argument parsing.

The language containers will use the containers module.

codecov · 2025-10-07T19:59:44Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.39%. Comparing base (c1b4570) to head (c0f6ba7).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2489      +/-   ##
==========================================
+ Coverage   85.62%   86.39%   +0.76%     
==========================================
  Files         102      102              
  Lines       10403    10587     +184     
==========================================
+ Hits         8908     9147     +239     
+ Misses       1168     1122      -46     
+ Partials      327      318       -9

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

suztomo · 2025-10-07T20:01:58Z

Test passed.

suztomo · 2025-10-07T20:02:28Z

@codyoss Would you review package structure in this new "containers" module?

suztomo · 2025-10-08T14:34:35Z

@meltsufin As talked yesterday, for each module in google-cloud-go 's librariangen, I put the reasons for include/exclude them in the pull request description.

suztomo · 2025-10-08T15:04:03Z

@quartzmo @julieqiu I appreciate if you can also give feedback on this module and package placement.

containers/bootstrap/bootstrap.go

codyoss

Overall I like the idea of this package but I think it would be useful to have a design of the public API.

codyoss · 2025-10-10T18:14:38Z

languagecontainer/go.mod

+
+go 1.25.0
+
+require github.com/googleapis/librarian v0.0.0


I think we should try to avoid depending on the other module if possibile.

With the comment you wrote for the "message" package, I'm now thinking of removing this module and having the language containers to depend on the github.com/googleapis/librarian module. The "languagecontainer/*" are packages inside the module that they use for language-container specific work.

It's now in go/sdk-librarian-containers-in-go?tab=t.ft3dwbukyu66

codyoss · 2025-10-10T18:15:45Z

languagecontainer/go.mod

+
+require github.com/googleapis/librarian v0.0.0
+
+replace github.com/googleapis/librarian => ../


I would avoid putting replace statement in a module other ppl will depend on. It can create cases where things will build here locally with CI but fail for ppl that depend on the library.

I'm now thinking of using the root librarian module for sharing with language containers. PTAL go/sdk-librarian-containers-in-go?tab=t.ft3dwbukyu66

codyoss · 2025-10-10T18:16:34Z

languagecontainer/bazel/parser.go

+// Package bazel provides functions to parse Bazel-related files.
+package bazel
+
+// TODO: Implement this package. Reference https://github.com/googleapis/google-cloud-go/tree/main/internal/librariangen/bazel


nit: don't add an empty package. We can add later when we know what we want this to look like.

codyoss · 2025-10-10T18:18:04Z

languagecontainer/cmd/cmd.go

+// generate-request.json and generate-response.json.
+// https://github.com/googleapis/librarian/blob/main/doc/language-onboarding.md#generate
+type GenerateFlags struct {
+	Librarian string


nit: add godoc to an public struct fields.

Let me focus on the package organization for now.

codyoss · 2025-10-10T18:23:38Z

languagecontainer/cmd/cmd.go

+
+// LanguageContainerFunctions lists the functions a language container
+// defines to accept requests from Librarian CLI.
+type LanguageContainerFunctions struct {


nit: I would just call this LanguageContainer

and or move all of these things up a directory and call Functions. That would give usage the look: languagecontainer.Functions in calling code.

That's a great idea. I added that in go/sdk-librarian-containers-in-go?tab=t.ft3dwbukyu66 as languagecontainer.Functions.

codyoss · 2025-10-10T18:28:09Z

languagecontainer/cmd/cmd.go

+//	}
+//
+// ```
+func LanguageContainerMain(args []string, functions LanguageContainerFunctions) {


nit: Run. If you like the other feedback about that would make it languagecontainer.Run(...)

Yes, I took that idea in go/sdk-librarian-containers-in-go?tab=t.ft3dwbukyu66.

codyoss · 2025-10-10T18:31:49Z

languagecontainer/execv/execv.go

+	cmd := exec.CommandContext(ctx, args[0], args[1:]...)
+	cmd.Env = os.Environ()
+	cmd.Dir = workingDir
+	slog.Debug("librariangen: running command", "command", strings.Join(cmd.Args, " "), "dir", cmd.Dir)


nit: remove all references to librariangen here. This is an idiom usually only used in error msgs and signals where an error came from. For logging you can see the source line if that feature is turned on. ( It is off by default)

Let's forget about this execv package for now to focus on the package organization.

codyoss · 2025-10-10T18:34:56Z

languagecontainer/execv/execv.go

+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package execv


I think the API for such a package should just mimic https://pkg.go.dev/os/exec#Cmd. It would be nice it was a drop in replacement for the stdlib -- API wise.

codyoss · 2025-10-10T18:37:18Z

languagecontainer/message/message.go

+
+// Package message declares the definition of the messages which Librarian
+// CLI and the language containers read and write.
+package message


If we do create a package like this I almost think a good goal here would be to invert the dep. The CLI should depend on this package. Although it is allowed since this in in the same monorepo, an outside package could never do what you did here by reaching into a Go internal directory.

I put the idea of creating a "message" package, outside the "internal" package. go/sdk-librarian-containers-in-go?tab=t.ft3dwbukyu66

codyoss · 2025-10-10T18:37:32Z

languagecontainer/protoc/protoc.go

+// limitations under the License.
+
+// Package protoc provides functions to run Protobuf compiler (protoc command)
+package protoc


Same a bazel, don't add it until define.

suztomo · 2025-10-10T19:44:51Z

Let me write a note about the plan.

suztomo

@codyoss @quartzmo PTAL go/sdk-librarian-containers-in-go?tab=t.ft3dwbukyu66. I'll close this pull request for now.

suztomo · 2025-10-10T20:43:36Z

languagecontainer/go.mod

+
+require github.com/googleapis/librarian v0.0.0
+
+replace github.com/googleapis/librarian => ../


I'm now thinking of using the root librarian module for sharing with language containers. PTAL go/sdk-librarian-containers-in-go?tab=t.ft3dwbukyu66

suztomo · 2025-10-10T20:43:49Z

languagecontainer/go.mod

+
+go 1.25.0
+
+require github.com/googleapis/librarian v0.0.0


It's now in go/sdk-librarian-containers-in-go?tab=t.ft3dwbukyu66

suztomo · 2025-10-10T20:44:19Z

languagecontainer/execv/execv.go

+	cmd := exec.CommandContext(ctx, args[0], args[1:]...)
+	cmd.Env = os.Environ()
+	cmd.Dir = workingDir
+	slog.Debug("librariangen: running command", "command", strings.Join(cmd.Args, " "), "dir", cmd.Dir)


Let's forget about this execv package for now to focus on the package organization.

suztomo · 2025-10-10T20:45:02Z

languagecontainer/cmd/cmd.go

+//	}
+//
+// ```
+func LanguageContainerMain(args []string, functions LanguageContainerFunctions) {


Yes, I took that idea in go/sdk-librarian-containers-in-go?tab=t.ft3dwbukyu66.

suztomo · 2025-10-10T20:45:36Z

languagecontainer/cmd/cmd.go

+
+// LanguageContainerFunctions lists the functions a language container
+// defines to accept requests from Librarian CLI.
+type LanguageContainerFunctions struct {


That's a great idea. I added that in go/sdk-librarian-containers-in-go?tab=t.ft3dwbukyu66 as languagecontainer.Functions.

suztomo · 2025-10-10T20:46:40Z

languagecontainer/cmd/cmd.go

+
+// Package cmd provides the main function LanguageContainerMain that
+// handles command line argument parsing and invocation of the corresponding methods.
+package cmd


Yes, let's do that. I like languagecontainer.Run idea. go/sdk-librarian-containers-in-go?tab=t.ft3dwbukyu66

suztomo · 2025-10-10T20:47:08Z

languagecontainer/cmd/cmd.go

+// generate-request.json and generate-response.json.
+// https://github.com/googleapis/librarian/blob/main/doc/language-onboarding.md#generate
+type GenerateFlags struct {
+	Librarian string


Let me focus on the package organization for now.

suztomo · 2025-10-10T20:47:52Z

languagecontainer/message/message.go

+
+// Package message declares the definition of the messages which Librarian
+// CLI and the language containers read and write.
+package message


I put the idea of creating a "message" package, outside the "internal" package. go/sdk-librarian-containers-in-go?tab=t.ft3dwbukyu66

feat: the containers module to share data and functions

94d4477

The language containers will use the containers module.

suztomo force-pushed the containers_module branch from fbdb3dd to 94d4477 Compare October 7, 2025 14:31

Adding bootstrap package

c632576

suztomo force-pushed the containers_module branch 4 times, most recently from de4b626 to 98a96c1 Compare October 7, 2025 18:26

Adding more package declarations

3e0d668

suztomo force-pushed the containers_module branch from 98a96c1 to 3e0d668 Compare October 7, 2025 18:28

suztomo added 4 commits October 7, 2025 19:17

Added bootstrap_test

06b8d33

Adding comments for the exported data

91a8370

Adding tests

b06d54e

exclude header check

4fb911c

suztomo requested a review from codyoss October 7, 2025 19:59

suztomo marked this pull request as ready for review October 7, 2025 20:02

suztomo requested a review from a team as a code owner October 7, 2025 20:02

suztomo requested review from julieqiu and quartzmo October 8, 2025 15:02

suztomo changed the title ~~feat: the containers module to share data and functions~~ feat: the containers module to share data and functions with language containers Oct 8, 2025

quartzmo reviewed Oct 9, 2025

View reviewed changes

containers/bootstrap/bootstrap.go Outdated Show resolved Hide resolved

suztomo added 5 commits October 10, 2025 14:27

Merge remote-tracking branch 'upstream/main' into containers_module

ec4c803

bootstrap package => cmd package

1e6cc25

adding cmd package

caeeb19

Reverted the use of interface

ef131ab

containers => languagecontainers

653fb85

suztomo changed the title ~~feat: the containers module to share data and functions with language containers~~ feat: the languagecontainer module to share data and functions with language containers Oct 10, 2025

suztomo added 2 commits October 10, 2025 15:07

update the path in test

9845ffa

Adding back the files

c0f6ba7

codyoss reviewed Oct 10, 2025

View reviewed changes

suztomo commented Oct 10, 2025

View reviewed changes

suztomo closed this Oct 10, 2025


		go 1.25.0

		require github.com/googleapis/librarian v0.0.0


		require github.com/googleapis/librarian v0.0.0

		replace github.com/googleapis/librarian => ../

Comments

Conversation

suztomo commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The languagecontainer module and packages

Language Container's View

Tests

Uh oh!

codecov bot commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

suztomo commented Oct 7, 2025

Uh oh!

suztomo commented Oct 7, 2025

Uh oh!

suztomo commented Oct 8, 2025

Uh oh!

suztomo commented Oct 8, 2025

Uh oh!

Uh oh!

codyoss left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

suztomo Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

suztomo commented Oct 10, 2025

Uh oh!

suztomo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

suztomo commented Oct 6, 2025 •

edited

Loading

codecov bot commented Oct 7, 2025 •

edited

Loading

suztomo Oct 10, 2025 •

edited

Loading